Stock Price Forecasting¶

Exploring the Time Series Problem of stock market forecasting.

All Datasets available @

BTC, AAPL, MSFT, TSLA, ^IXIC(NASDAQ), ^BVSP(IBOVESPA):
https://finance.yahoo.com/

S&P 500:
https://www.kaggle.com/datasets/andrewmvd/sp-500-stocks?select=sp500_index.csv

Group Members:

200028880 - Wallace Ben Teng Lin Wu

222011561 - Mateus Elias de Macedo

222011525 - Erick Hideki Taira

221029051 - Rodrigo Marques Maranhao

Preprocessing¶

In [1]:
import pandas as pd
import datetime

def str_to_datetime(s):
    """ Converts a string object to the respective datetime object"""

    year, month, day = [int(i) for i in s.split('-')]
    return datetime.datetime(year=year, month=month, day=day)


price_dict = {
    "Adj Close" : "Price",
    "S&P500" : "Price",
}

def load_df(filename):
    """
    Create a pandas dataframe, filter to leave only the Price column,
    convert date to datetime and make it the index
    """

    df = pd.read_csv(filename)
    df.rename(columns = price_dict, inplace = True)

    # Univariate analysis
    df = df[["Date", "Price"]]

    # Convert date type objects to datetime object
    df["Date"] = df["Date"].apply(str_to_datetime)

    # Turn "Date" Column into dataframe index
    df.index = df.pop("Date")

    return df.dropna()


df = load_df("Datasets/MSFT.csv")
In [2]:
df
Out[2]:
Price
Date
1986-03-13 0.060396
1986-03-14 0.062553
1986-03-17 0.063632
1986-03-18 0.062014
1986-03-19 0.060936
... ...
2023-11-01 346.070007
2023-11-02 348.320007
2023-11-03 352.799988
2023-11-06 356.529999
2023-11-07 360.529999

9491 rows × 1 columns

In [3]:
import matplotlib.pyplot as plt

plt.figure(figsize=(10,6))
plt.plot(df.index, df["Price"])
plt.title("Full Dataset")
plt.show()
No description has been provided for this image
In [4]:
# Choose the amount of days to consider from the dataset
days = 5000 # ~13 years

# numbers of days to consider in the input of the model
lookback = 15 #


def df_to_windowed(fullDF, n=lookback, daysSelected=days):
    """
    Create a windowed Dataframe (converting into a supervised problem).
    Therefore, the last {lookback} days prices will be the (input)
    and will generate the next day price (output)
    """

    tmp_df = pd.DataFrame()
    for i in range(n, 0, -1):
        tmp_df[f"Last-{i} Price"] = fullDF["Price"].shift(periods=i)
    tmp_df["Price"] = fullDF["Price"]

    return tmp_df.dropna()[-daysSelected:]


windowed_df = df_to_windowed(df)
In [5]:
windowed_df
Out[5]:
Last-15 Price Last-14 Price Last-13 Price Last-12 Price Last-11 Price Last-10 Price Last-9 Price Last-8 Price Last-7 Price Last-6 Price Last-5 Price Last-4 Price Last-3 Price Last-2 Price Last-1 Price Price
Date
2003-12-29 16.282078 16.445030 16.532766 16.664377 16.676914 16.701979 16.758381 16.958931 16.946400 17.172014 17.146950 17.034134 17.015335 16.946400 17.052935 17.209623
2003-12-30 16.445030 16.532766 16.664377 16.676914 16.701979 16.758381 16.958931 16.946400 17.172014 17.146950 17.034134 17.015335 16.946400 17.052935 17.209623 17.247215
2003-12-31 16.532766 16.664377 16.676914 16.701979 16.758381 16.958931 16.946400 17.172014 17.146950 17.034134 17.015335 16.946400 17.052935 17.209623 17.247215 17.153221
2004-01-02 16.664377 16.676914 16.701979 16.758381 16.958931 16.946400 17.172014 17.146950 17.034134 17.015335 16.946400 17.052935 17.209623 17.247215 17.153221 17.203354
2004-01-05 16.676914 16.701979 16.758381 16.958931 16.946400 17.172014 17.146950 17.034134 17.015335 16.946400 17.052935 17.209623 17.247215 17.153221 17.203354 17.635786
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2023-11-01 332.420013 331.160004 327.730011 332.640015 332.059998 330.109985 331.320007 326.670013 329.320007 330.529999 340.670013 327.890015 329.809998 337.309998 338.109985 346.070007
2023-11-02 331.160004 327.730011 332.640015 332.059998 330.109985 331.320007 326.670013 329.320007 330.529999 340.670013 327.890015 329.809998 337.309998 338.109985 346.070007 348.320007
2023-11-03 327.730011 332.640015 332.059998 330.109985 331.320007 326.670013 329.320007 330.529999 340.670013 327.890015 329.809998 337.309998 338.109985 346.070007 348.320007 352.799988
2023-11-06 332.640015 332.059998 330.109985 331.320007 326.670013 329.320007 330.529999 340.670013 327.890015 329.809998 337.309998 338.109985 346.070007 348.320007 352.799988 356.529999
2023-11-07 332.059998 330.109985 331.320007 326.670013 329.320007 330.529999 340.670013 327.890015 329.809998 337.309998 338.109985 346.070007 348.320007 352.799988 356.529999 360.529999

5000 rows × 16 columns

In [6]:
windowed_df["Price"].describe()
Out[6]:
count    5000.000000
mean       80.652762
std        92.782729
min        11.327569
25%        20.142975
50%        30.797449
75%       102.435174
max       360.529999
Name: Price, dtype: float64
In [7]:
def split_xy(windowedNP):
    """
    Split np.array into X and y
    """

    X = windowedNP[:, :-1]
    y = windowedNP[:, -1]
    return (X, y)

Standardization (Padronização)¶

Standardization != Normalization

Padronização != Normalização

In [8]:
from IPython import display
display.Image("Images/Standardization.png")
Out[8]:
No description has been provided for this image
In [9]:
from sklearn.preprocessing import StandardScaler

def scale_data(train, vali, test):
    """ Get Scaled Data """ 
    
    scaler = StandardScaler()
    X_train, y_train = split_xy(scaler.fit_transform(train))
    X_vali, y_vali = split_xy(scaler.transform(vali))
    X_test, y_test = split_xy(scaler.transform(test))
    return scaler, [X_train, X_vali, X_test], [y_train, y_vali, y_test]
In [10]:
def descale_data(train, vali, test, pred, scaler):
    """ Get de-Scaled Data """ 
    X_train, y_train = split_xy(train.to_numpy())
    X_vali, y_vali = split_xy(vali.to_numpy())
    X_test, y_test = split_xy(test.to_numpy())
    X_result, y_result = split_xy(scaler.inverse_transform(pred))
    return [y_train, y_vali, y_test, y_result]

Models¶

Theory¶

1 Dimensional Convolution:

No description has been provided for this image
No description has been provided for this image

1D Max Pooling:

In [12]:
display.Image("Images/MaxPooling.png")
Out[12]:
No description has been provided for this image
In [13]:
display.Image("Images/Conv+Max.png")

# Max_pooling aceita padding também!
Out[13]:
No description has been provided for this image

Long Short Term Memory (LSTM):

In [14]:
display.Image("Images/LSTM.png")
Out[14]:
No description has been provided for this image
In [15]:
display.Image("Images/Forget_Gate.png")
Out[15]:
No description has been provided for this image
In [16]:
display.Image("Images/Input_Gate.png")
Out[16]:
No description has been provided for this image
In [17]:
display.Image("Images/Output_Gate.png")
Out[17]:
No description has been provided for this image

Dropout Layer:

No description has been provided for this image

Best Models¶

That survived my manual natural selection...

In [18]:
from tensorflow.keras.models import Sequential
from tensorflow.keras import layers

# model input: (last {lookback} days prices, 1 feature = "price")
models = []
In [19]:
models.append(
    Sequential([ # CNN+LSTM+Dropout
       layers.Input((lookback, 1)),
       layers.Conv1D(128, kernel_size=3, activation="relu", padding="same"),
       layers.MaxPooling1D(pool_size=2, padding="same"),
       layers.LSTM(128, return_sequences=True),
       layers.Flatten(),
       layers.Dropout(0.3),
       layers.Dense(128),
       layers.Dense(1)
    ]),
)
models[-1].summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv1d (Conv1D)             (None, 15, 128)           512       
                                                                 
 max_pooling1d (MaxPooling1  (None, 8, 128)            0         
 D)                                                              
                                                                 
 lstm (LSTM)                 (None, 8, 128)            131584    
                                                                 
 flatten (Flatten)           (None, 1024)              0         
                                                                 
 dropout (Dropout)           (None, 1024)              0         
                                                                 
 dense (Dense)               (None, 128)               131200    
                                                                 
 dense_1 (Dense)             (None, 1)                 129       
                                                                 
=================================================================
Total params: 263425 (1.00 MB)
Trainable params: 263425 (1.00 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [20]:
models.append(
    Sequential([ # LSTM+Dropout
        layers.Input((lookback, 1)),
        layers.LSTM(128, return_sequences=False),
        layers.Dropout(0.3),
        layers.Dense(128),
        layers.Dense(128),
        layers.Dense(1)
    ]),
)
models[-1].summary()
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 lstm_1 (LSTM)               (None, 128)               66560     
                                                                 
 dropout_1 (Dropout)         (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 128)               16512     
                                                                 
 dense_3 (Dense)             (None, 128)               16512     
                                                                 
 dense_4 (Dense)             (None, 1)                 129       
                                                                 
=================================================================
Total params: 99713 (389.50 KB)
Trainable params: 99713 (389.50 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [21]:
models.append(
    Sequential([ # CNN
        layers.Input((lookback, 1)),
        layers.Conv1D(128, kernel_size=3, activation="relu", padding="same"),
        layers.MaxPooling1D(pool_size=2, padding="same"),
        layers.Flatten(),
        layers.Dense(128),
        layers.Dense(1)
    ]),
)
models[-1].summary()
Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv1d_1 (Conv1D)           (None, 15, 128)           512       
                                                                 
 max_pooling1d_1 (MaxPoolin  (None, 8, 128)            0         
 g1D)                                                            
                                                                 
 flatten_1 (Flatten)         (None, 1024)              0         
                                                                 
 dense_5 (Dense)             (None, 128)               131200    
                                                                 
 dense_6 (Dense)             (None, 1)                 129       
                                                                 
=================================================================
Total params: 131841 (515.00 KB)
Trainable params: 131841 (515.00 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
In [22]:
models.append(
    Sequential([ # Simple Neural Network
        layers.Input((lookback, 1)),
        layers.Flatten(),
        layers.Dense(128),
        layers.Dense(128),
        layers.Dense(1)
    ]),
)
models[-1].summary()
Model: "sequential_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 flatten_2 (Flatten)         (None, 15)                0         
                                                                 
 dense_7 (Dense)             (None, 128)               2048      
                                                                 
 dense_8 (Dense)             (None, 128)               16512     
                                                                 
 dense_9 (Dense)             (None, 1)                 129       
                                                                 
=================================================================
Total params: 18689 (73.00 KB)
Trainable params: 18689 (73.00 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Model Training¶

Auxilary Functions¶

In [23]:
# For each year, 60% train, 20% validation, 20% test
def sliding_window(windowed, trainSZ=2100, valiSZ=450, testSZ=450, step=20):
    """
    Sliding Window Generator
    """

    for i in range(0, len(windowed) - trainSZ - testSZ - valiSZ + 1, step):
        train_slice = windowed[i : i+trainSZ]
        vali_slice = windowed[i+trainSZ : i+trainSZ+valiSZ]
        test_slice = windowed[i+trainSZ+valiSZ : i+trainSZ+valiSZ+testSZ]
        yield (train_slice, vali_slice, test_slice)
In [64]:
windows_list = list(sliding_window(windowed_df))
windows_cnt = len(windows_list)

plt.figure(figsize=(10,6))
plt.title(f"Number of Selected Windows: {windows_cnt}")
plt.legend([
    "Training Observations",
    "Validation Observations",
    "Testing Observations",
])
for train, vali, test in windows_list:
    plt.axvline(train.index[0], color="tab:gray")
    plt.plot(train.index, train["Price"], color="tab:blue")
    plt.plot(vali.index, vali["Price"], color="tab:orange")
    plt.plot(test.index, test["Price"], color="tab:green")
    plt.axvline(test.index[-1], color="tab:gray")

plt.show()
No description has been provided for this image
In [66]:
plt.figure(figsize=(10,6))
train, vali, test = windows_list[-1]
plt.axvline(train.index[0], color="tab:gray")
plt.plot(train.index, train["Price"], color="tab:blue")
plt.plot(vali.index, vali["Price"], color="tab:orange")
plt.plot(test.index, test["Price"], color="tab:green")
plt.axvline(test.index[-1], color="tab:gray")
plt.title(f"Last Window: # {windows_cnt-1}")
plt.legend([
    "Training Observations",
    "Validation Observations",
    "Testing Observations",
])
plt.show()
No description has been provided for this image
In [25]:
import numpy as np
from sklearn.metrics import confusion_matrix

def compute_accuracy_and_cm(y_val, y_test, y_pred):
    """
    Computes the accuracy score and the confusion matrix
    For simplicity, zero price change are considered as positive
    """

    sz = len(y_test)
    y_ref = np.append(y_val[-1], y_test)
    
    y_test_label = np.zeros(sz)
    y_pred_label = np.zeros(sz)

    acc = 0
    for i in range(sz):
        y_test_label[i] = 1 if ((y_test[i] - y_ref[i]) >= 0) else -1
        y_pred_label[i]  = 1 if ((y_pred[i] - y_ref[i]) >= 0) else -1

        if y_test_label[i] == y_pred_label[i]:
            acc += 1

    cm = confusion_matrix(y_true=y_test_label, y_pred=y_pred_label)
    return acc/sz, cm
In [26]:
from matplotlib import patches
patienceSelected = 50

def plot_loss_curve(history, model_idx, i, patience=patienceSelected):
    """
    Plots the loss curve for the model fitting process
    """
    
    logs = history.history
    aux_list = [(val, i) for i, val in enumerate(logs['combine_metric'])]
    best = min(aux_list)
    last = len(logs['combine_metric'])

    plt.figure(figsize=(10,6))
    plt.title(f"Loss Curve for: Model {model_idx}, Window {i}")
    plt.plot(logs["loss"], label="Training Loss")
    plt.plot(logs["val_loss"], label="Validation Loss")
    plt.plot(logs["combine_metric"], label="Combined Loss")
    plt.ylabel("Loss")
    plt.xlabel("Epoch")

    plt.axvline(last-1, color="tab:gray", ymax=0.3, linestyle='--')
    plt.axvline(last-patience-1, color="tab:gray", ymax=0.3, linestyle='--')
    plt.axvline(best[1], color="tab:red", ymax=0.3, linestyle='--')
    
    red_patch = patches.Patch(
        color="tab:red", 
        label=f"best epoch={best[1]}")
    
    gray_patch = patches.Patch(
        color="tab:gray", 
        label=f"Early Stop Limits ({last-patience-1}, {last-1})")

    handles, labels = plt.gca().get_legend_handles_labels()
    handles.extend([red_patch, gray_patch])

    plt.legend(handles=handles, loc="upper right")
    plt.show()
In [27]:
def plot_predictions(dates, ys, metrics, model_idx, i):
    """
    Plots the predicted curve, comparing with observation data
    """
    
    dates_train, dates_vali, dates_test = dates
    y_train, y_vali, y_test, y_result = ys
    rmse, mae, mape, r2, acc = metrics
    
    plt.figure(figsize=(10,6))
    plt.plot(dates_train, y_train)
    plt.plot(dates_vali, y_vali)
    plt.plot(dates_test, y_test)
    plt.plot(dates_test, y_result)
    plt.legend([
        "Training Observations",
        "Validation Observations",
        "Testing Observations",
        "Testing Predictions"
    ])
    plt.title(f"Model {model_idx}, Window {i} \n \
              RMSE={rmse:.3f}, MAE={mae:.3f}, MAPE={mape:.3f}, R2={r2:.3f}" )
    plt.show()
In [28]:
from sklearn.metrics import ConfusionMatrixDisplay

def plot_confusion_matrix(cm, metrics, model_idx, i):
    """
    Plots the confusion matrix for the price change classification
    """

    rmse, mae, mape, r2, acc = metrics

    cm_plt = ConfusionMatrixDisplay(cm, 
                                    display_labels=["Positive", "Negative"])
    cm_plt.plot()
    cm_plt.ax_.set(
        title= f"Model {model_idx}, Window {i}, Accuracy={acc:.3f}",
        xlabel= "Predicted Price Change",
        ylabel= "Actual Price Change"
    )
    plt.show()

Creating important callbacks that will be used in the model training to avoid overfitting and also redundant longer training:¶

In [29]:
from keras.callbacks import EarlyStopping , Callback, ModelCheckpoint
import h5py 

class CombineCallback(Callback):
    def __init__(self, **kargs):
        super(CombineCallback, self).__init__(**kargs)
    def on_epoch_end(self, epoch, logs={}):
        f = 0.2 # f=vali_factor, 80% training loss, 20% validation loss
        logs['combine_metric'] = f*logs['val_loss']+(1-f)*logs['loss']

combined_cb = CombineCallback()
model_checkpoint = ModelCheckpoint(
    filepath="Models/tmp_best_model.h5", 
    monitor="combine_metric", 
    mode="min", 
    save_best_only=True, 
    save_weights_only=True,
    verbose=False
)
earlyStop = EarlyStopping(monitor="combine_metric", 
                          min_delta=0, 
                          patience=patienceSelected, 
                          mode="min", 
                          verbose=False)

Main Function¶

In [30]:
from sklearn.metrics import mean_squared_error, mean_absolute_error
from sklearn.metrics import r2_score, mean_absolute_percentage_error
from tensorflow.keras.optimizers import Adam

def cross_validation(model, generator, model_idx, flag_plot=0):
    """
    Performs Cross validation for all models and all sliding windows

    Calculates the cross validation score:
        ("RMSE", "MAE", "MAPE", "R2", "Accuracy");

    Accuracy is computed by classifying if the relative price change 
    for day i was positive or negative
    """

    cv_score = pd.DataFrame(columns=["RMSE", "MAE", "MAPE", "R2", "Acc"])

    for i, (train, vali, test) in enumerate(generator):
        # Get Dates = [dates_train, dates_vali, dates_test]
        dates = [i.index for i in [train, vali, test]]

        # Scale data
        scaler, X_sc, y_sc = scale_data(train, vali, test)
        X_train_sc, X_vali_sc, X_test_sc = X_sc
        y_train_sc, y_vali_sc, y_test_sc = y_sc

        # Fit, save best model and Predict
        model.load_weights("Models/empty_model.h5", 
                           skip_mismatch=True, by_name=True)
        
        model.reset_states()
        history = model.fit(
            X_train_sc, y_train_sc,
            validation_data=(X_vali_sc, y_vali_sc),
            epochs=200, # maximum number of epochs
            batch_size=64, # better for jumping local minimas
            verbose=False,
            callbacks=[combined_cb, earlyStop, model_checkpoint]
        )
        model.load_weights("Models/tmp_best_model.h5",
                           skip_mismatch=True, by_name=True)
        
        preds_sc = model.predict(X_test_sc, verbose=False)

        # Descale data
        stacked_pred = np.hstack((X_test_sc, preds_sc))
        ys =  descale_data(train, vali, test, stacked_pred, scaler)
        [y_train, y_vali, y_test, y_result] = ys

        # Compute Metrics
        rmse = mean_squared_error(y_test, y_result, squared=False)
        mae = mean_absolute_error(y_test, y_result)
        mape = mean_absolute_percentage_error(y_test, y_result)
        r2 = r2_score(y_test, y_result)
        acc, cm = compute_accuracy_and_cm(y_vali, y_test, y_result)

        metrics = [rmse, mae, mape, r2, acc]

        # Plot All Curves and Metrics; Also loss curves
        if flag_plot == 2:
            plot_loss_curve(history, model_idx, i)
            plot_predictions(dates, ys, metrics, model_idx, i)
            plot_confusion_matrix(cm, metrics, model_idx, i)

        # Plot only 5 Curves and Metrics;
        elif (flag_plot == 1 and (i % (windows_cnt//5)) == 0):
            plot_loss_curve(history, model_idx, i)
            plot_predictions(dates, ys, metrics, model_idx, i)
            plot_confusion_matrix(cm, metrics, model_idx, i)

        # Append Result
        cv_score.loc[len(cv_score)] = metrics

    return cv_score
In [31]:
# For each model, perform a cross validation training,
# plot graphs and compute metrics if wanted
cv_scores = []

def run_model(model, i):
    """ 
    Creates the model with the selected architecture and empty weights,
    compile it with mse loss function and Adam optimizer
    and computes the cross validation results for this model
    """

    model.compile(
        loss="mean_squared_error",
        optimizer=Adam(learning_rate=0.0001)
    )   
    model.save_weights("Models/empty_model.h5")
    generator = sliding_window(windowed_df)
    cv_score = cross_validation(model, generator, i, 1)
    cv_scores.append(cv_score)

Backpropagation Method: Adam Optimizer

"We propose Adam, a method for efficient stochastic optimization that only requires first-order gradients with little memory requirement. The method computes individual adaptive learning rates for
different parameters from estimates of first and second moments of the gradients; the name Adam
is derived from adaptive moment estimation. Our method is designed to combine the advantages
of two recently popular methods: AdaGrad (Duchi et al., 2011), which works well with sparse gradients, and RMSProp (Tieleman & Hinton, 2012), which works well in on-line and non-stationary
settings; 

Some of Adam’s advantages are that the magnitudes of parameter updates are invariant to
rescaling of the gradient, its stepsizes are approximately bounded by the stepsize hyperparameter,
it does not require a stationary objective, it works with sparse gradients, and it naturally performs a
form of step size annealing."

~Diederik P. Kingma, Jimmy Lei Ba (Authors)

No description has been provided for this image

ADAM: Adaptive moment estimation

Adaptive momentum and adaptive learning rate !!

No description has been provided for this image No description has been provided for this image
In [32]:
# CNN + LSTM + Dropout
run_model(models[0], 0)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [33]:
# LSTM + Dropout
run_model(models[1], 1)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [34]:
# CNN
run_model(models[2], 2)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
In [35]:
# Simple Neural Network
run_model(models[3], 3)
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image
No description has been provided for this image

Results¶

In [36]:
# Output summary (mean, std, min, max)
for i, cv_score in enumerate(cv_scores):
    print(f"Model {i}")
    print(cv_score.describe(), "\n\n")
Model 0
             RMSE         MAE        MAPE          R2         Acc
count  101.000000  101.000000  101.000000  101.000000  101.000000
mean     3.895234    3.075236    0.021306    0.961728    0.524180
std      2.702409    2.180319    0.009673    0.024796    0.039309
min      0.683601    0.497610    0.008261    0.878070    0.428889
25%      1.149078    0.918554    0.015178    0.941309    0.502222
50%      4.456008    3.609054    0.017775    0.965491    0.524444
75%      5.662584    4.461142    0.027819    0.984983    0.557778
max     10.349789    8.664503    0.043119    0.995182    0.588889 


Model 1
             RMSE         MAE        MAPE          R2         Acc
count  101.000000  101.000000  101.000000  101.000000  101.000000
mean     9.628349    7.874803    0.045159    0.833011    0.459978
std      8.790591    7.196723    0.021031    0.137579    0.030172
min      0.704930    0.502852    0.010714    0.339006    0.411111
25%      2.339971    1.770794    0.027120    0.804450    0.433333
50%      7.281628    6.114446    0.044539    0.876047    0.457778
75%     14.199059   11.176512    0.054969    0.922518    0.486667
max     31.357097   26.573715    0.095304    0.984503    0.517778 


Model 2
             RMSE         MAE        MAPE          R2         Acc
count  101.000000  101.000000  101.000000  101.000000  101.000000
mean     2.699445    2.034105    0.015535    0.975658    0.506491
std      1.663646    1.269412    0.005354    0.027713    0.042541
min      0.642425    0.460840    0.008205    0.781972    0.411111
25%      1.351657    1.027763    0.012711    0.969669    0.480000
50%      1.928585    1.610527    0.014282    0.986385    0.506667
75%      4.011799    2.981743    0.016484    0.990132    0.546667
max      5.880076    4.651696    0.042051    0.995010    0.568889 


Model 3
             RMSE         MAE        MAPE          R2         Acc
count  101.000000  101.000000  101.000000  101.000000  101.000000
mean     2.695449    2.016152    0.014529    0.980740    0.497514
std      1.729265    1.309784    0.003360    0.013720    0.052106
min      0.615531    0.406209    0.007916    0.915134    0.402222
25%      0.959838    0.741015    0.012418    0.973966    0.453333
50%      1.946378    1.578985    0.014739    0.985687    0.493333
75%      4.170105    3.055891    0.015928    0.988714    0.540000
max      5.730381    4.524243    0.026676    0.995530    0.584444 


In [67]:
# Section by Parameter
metrics = ["RMSE", "MAE", "MAPE", "R2", "Acc"]

evaluation = {
    "RMSE": {},
    "MAE": {},
    "MAPE": {},
    "R2": {},
    "Acc": {},
}

for model_idx, cv_score in enumerate(cv_scores):
    for param in metrics:
        evaluation[param][f"Model {model_idx}"] = cv_score[param].mean()

def plot_metric(param, logScale=False):
    """
    Plots the evaluation metrics, comparing each model
    """

    if logScale:
        plt.figure(figsize=(10,6))
        plt.title(param + " in Log Scale")
        plt.bar(list(evaluation[param].keys()), 
                list(evaluation[param].values()), 
                color="tab:orange")
        plt.xlabel("Models")
        plt.ylabel("Metrics Value")
        plt.yscale('log')
        plt.show()

    else:
        plt.figure(figsize=(10,6))
        plt.title(param)
        plt.bar(list(evaluation[param].keys()), 
                list(evaluation[param].values()))
        plt.xlabel("Models")
        plt.ylabel("Metrics Value")
        plt.show()
In [38]:
display.Image("Images/RootMeanSquareError.png")
Out[38]:
No description has been provided for this image
In [39]:
plot_metric("RMSE")
No description has been provided for this image
In [40]:
plot_metric("RMSE", 1)
No description has been provided for this image
In [41]:
display.Image("Images/MeanAbsoluteError.png")
Out[41]:
No description has been provided for this image
In [42]:
plot_metric("MAE")
No description has been provided for this image
In [43]:
plot_metric("MAE", 1)
No description has been provided for this image
In [44]:
display.Image("Images/MeanAbsolutePercentageError.png")
Out[44]:
No description has been provided for this image
In [45]:
plot_metric("MAPE")
No description has been provided for this image
In [46]:
plot_metric("MAPE", 1)
No description has been provided for this image
In [47]:
display.Image("Images/R2-DeterminationCoefficient.png")
Out[47]:
No description has been provided for this image
In [48]:
plot_metric("R2")
No description has been provided for this image
In [49]:
plot_metric("R2", 1)
No description has been provided for this image
In [50]:
display.Image("Images/Accuracy.png")
Out[50]:
No description has been provided for this image
In [51]:
plot_metric("Acc")
No description has been provided for this image
In [52]:
plot_metric("Acc", 1)
No description has been provided for this image
In [53]:
# Output complete results   
for i, cv_score in enumerate(cv_scores):
    print(f"Model {i}")
    print(cv_score)
Model 0
         RMSE       MAE      MAPE        R2       Acc
0    0.787477  0.577116  0.014655  0.950195  0.515556
1    0.961965  0.735219  0.018258  0.937613  0.526667
2    1.025013  0.775381  0.018874  0.934461  0.526667
3    1.065233  0.807187  0.019402  0.928009  0.524444
4    1.103644  0.843336  0.020038  0.923561  0.526667
..        ...       ...       ...       ...       ...
96   5.897175  4.710589  0.017182  0.968836  0.491111
97   5.945330  4.745758  0.017228  0.971457  0.495556
98   5.960590  4.739940  0.017252  0.970987  0.495556
99   5.867653  4.672015  0.016994  0.971648  0.500000
100  5.835192  4.667708  0.016916  0.973805  0.500000

[101 rows x 5 columns]
Model 1
          RMSE        MAE      MAPE        R2       Acc
0     1.335157   1.191497  0.030293  0.856827  0.493333
1     0.990914   0.788837  0.019502  0.933802  0.493333
2     0.862260   0.643464  0.015650  0.953622  0.495556
3     1.123366   0.930515  0.022504  0.919937  0.495556
4     1.392760   1.214147  0.028957  0.878267  0.491111
..         ...        ...       ...       ...       ...
96    8.098891   6.673524  0.023479  0.941221  0.511111
97    8.752363   7.202692  0.025108  0.938142  0.508889
98    9.411969   7.682160  0.027711  0.927661  0.486667
99   12.457886  10.883138  0.037870  0.872195  0.511111
100   8.773446   7.266590  0.025326  0.940783  0.497778

[101 rows x 5 columns]
Model 2
         RMSE       MAE      MAPE        R2       Acc
0    0.788876  0.593824  0.015155  0.950018  0.493333
1    0.668899  0.460840  0.011601  0.969835  0.491111
2    1.074884  0.912324  0.022541  0.927929  0.493333
3    1.853801  1.723600  0.042051  0.781972  0.495556
4    0.699355  0.483527  0.011640  0.969306  0.502222
..        ...       ...       ...       ...       ...
96   5.713737  4.499804  0.016474  0.970744  0.500000
97   5.821043  4.595650  0.016711  0.972638  0.495556
98   5.880076  4.651696  0.016920  0.971766  0.500000
99   5.755412  4.556611  0.016575  0.972722  0.504444
100  5.690243  4.515196  0.016389  0.975090  0.511111

[101 rows x 5 columns]
Model 3
         RMSE       MAE      MAPE        R2       Acc
0    0.615531  0.406209  0.010443  0.969570  0.484444
1    1.066529  0.912504  0.022910  0.923313  0.493333
2    0.734260  0.531329  0.013139  0.966369  0.493333
3    0.909371  0.715635  0.017559  0.947535  0.504444
4    0.680336  0.464767  0.011244  0.970953  0.522222
..        ...       ...       ...       ...       ...
96   5.584017  4.387769  0.016055  0.972058  0.486667
97   5.685540  4.471663  0.016245  0.973897  0.480000
98   5.730381  4.524243  0.016431  0.973185  0.484444
99   5.622617  4.426225  0.016095  0.973966  0.484444
100  5.579112  4.401072  0.015972  0.976054  0.500000

[101 rows x 5 columns]